[Transform] QuIP Modifier #1648
Open
+195
−7
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Kyle Sayers <[email protected]>
Signed-off-by: Kyle Sayers <[email protected]>
Signed-off-by: Kyle Sayers <[email protected]>
Signed-off-by: Brian Dellabetta <[email protected]>
Signed-off-by: Brian Dellabetta <[email protected]>
Signed-off-by: Brian Dellabetta <[email protected]>
Signed-off-by: Brian Dellabetta <[email protected]>
Signed-off-by: Brian Dellabetta <[email protected]>
Signed-off-by: Kyle Sayers <[email protected]>
Signed-off-by: Kyle Sayers <[email protected]>
Signed-off-by: Kyle Sayers <[email protected]>
Signed-off-by: Kyle Sayers <[email protected]>
Signed-off-by: Kyle Sayers <[email protected]>
Signed-off-by: Kyle Sayers <[email protected]>
Signed-off-by: Kyle Sayers <[email protected]>
Signed-off-by: Kyle Sayers <[email protected]>
Signed-off-by: Kyle Sayers <[email protected]>
Signed-off-by: Kyle Sayers <[email protected]>
Signed-off-by: Kyle Sayers <[email protected]>
Signed-off-by: Kyle Sayers <[email protected]>
Signed-off-by: Kyle Sayers <[email protected]>
Signed-off-by: Kyle Sayers <[email protected]>
Signed-off-by: Kyle Sayers <[email protected]>
Signed-off-by: Kyle Sayers <[email protected]>
Signed-off-by: Kyle Sayers <[email protected]>
Signed-off-by: Kyle Sayers <[email protected]>
Signed-off-by: Kyle Sayers <[email protected]>
Signed-off-by: Kyle Sayers <[email protected]>
brian-dellabetta
force-pushed
the
kylesayrs/transform-quip-modifier
branch
from
7805321 to
ac7dbcd
Compare
|
Evals on Llama 3.2 1B with Quip (num_fewshot 8, limit 1000 to be compatible with results here) :
|
Signed-off-by: Kyle Sayers <[email protected]>
Signed-off-by: Kyle Sayers <[email protected]>
Signed-off-by: Kyle Sayers <[email protected]>
Signed-off-by: Kyle Sayers <[email protected]>
Signed-off-by: Kyle Sayers <[email protected]>
Signed-off-by: Kyle Sayers <[email protected]>
Signed-off-by: Kyle Sayers <[email protected]>
Signed-off-by: Kyle Sayers <[email protected]>
brian-dellabetta
previously approved these changes
Aug 6, 2025
dsikka
requested changes
Aug 7, 2025
| @@ -27,5 +28,9 @@ def __call__( | |||
| :param dataloader: loads data for calibration | |||
| :param dataset_args: dataset arguments relevant to pipelines | |||
| """ | |||
| # some ops are still performed on the model by modifiers | |||
| # we want those ops to occur on the GPU | |||
| dispatch_for_generation(model) | |||
There was a problem hiding this comment.
That means we can leverage more than one gpu for data free cases, including weight-only RTN schemes?
There was a problem hiding this comment.
Technically yes, although the weights are still calibrated in a synchronous for loop, so there's no speedup gained from the extra gpus
Signed-off-by: Kyle Sayers <[email protected]>
Signed-off-by: Kyle Sayers <[email protected]>
Signed-off-by: Kyle Sayers <[email protected]>
Signed-off-by: Kyle Sayers <[email protected]>
Signed-off-by: Kyle Sayers <[email protected]>
|
|
||
| # Configure the quantization algorithm to run. | ||
| # * apply spinquant transforms to model in order to make quantization easier | ||
| # * quantize the weights to 4 bit with GPTQ with a group size 128 |
There was a problem hiding this comment.
I had to update this in the spinquant example as well
Suggested change
| # * quantize the weights to 4 bit with GPTQ with a group size 128 | |
| # * quantize the weights to 4 bit and group size 128 |
|
|
||
| QuIP and QuIP# apply transforms to every linear layer, two of which are fused into | ||
| the model weights and two of which remain as online rotations computed at runtime. | ||
|
|
There was a problem hiding this comment.
Should add lifecycle here, can probably copy-paste from SpinQuantModifier
dsikka
approved these changes
Aug 13, 2025
| targets="Linear", scheme="FP8_BLOCK", ignore=["lm_head", "re:.*mlp.gate$"], | ||
| targets="Linear", | ||
| scheme="FP8_BLOCK", | ||
| ignore=["lm_head", "re:.*mlp.gate$"], |
There was a problem hiding this comment.
I think this was fixed in a previous PR. Maybe rebase
|
|
||
| # Configure the quantization algorithm to run. | ||
| # * apply spinquant transforms to model in order to make quantization easier | ||
| # * quantize the weights to 4 bit with GPTQ with a group size 128 |
There was a problem hiding this comment.
using GPTQ should still be there
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Purpose
Prerequisites
Changes
quip_example.pyto examples folderQuIPModifierwhich handles the construction of a quip-style transform configTesting
Evaluation
Evaluation performed by @brian-dellabetta
Evals on Llama 3.2 1B with Quip (num_fewshot 8, limit 1000 to be compatible with results here) :
Follow Ups